# Putting It All Together: Common Scenarios

Let's walk through complete real-world scenarios to see how everything we've learned works together. We'll follow the data step-by-step from user action to final result.

## Scenario 1: Voice Assistant Response

**User Story**: Alex asks "What's on my calendar today?" and sees their schedule on the glasses.

### Step 1: Audio Capture
Alex speaks into their glasses microphone.

```typescript
// Glasses → Cloud (via mobile app)
{
  type: "AUDIO_DATA",
  data: "base64encodedaudio...",
  format: "pcm",
  sampleRate: 16000
}
```

### Step 2: Audio Processing
The AudioManager in Alex's UserSession receives the audio.

```typescript
// In AudioManager
handleAudioData(audioData) {
  // Buffer the audio
  this.audioBuffer.push(audioData);
  
  // Forward to TranscriptionManager
  this.session.transcriptionManager.processAudio(audioData);
}
```

### Step 3: Transcription
TranscriptionManager sends audio to the transcription service.

```typescript
// TranscriptionManager → Deepgram
stream.write(audioData);

// Deepgram → TranscriptionManager
{
  transcript: "What's on my calendar today?",
  isFinal: true,
  confidence: 0.98
}
```

### Step 4: Event Distribution
The SubscriptionService finds apps subscribed to transcriptions.

```typescript
// In SubscriptionService
const subscribers = this.getSubscribers('transcription', sessionId);
// Returns: ['com.assistant.app', 'com.notes.app']

// Cloud → Assistant App
{
  type: "DATA_STREAM",
  streamType: "transcription",
  data: {
    type: "transcription",
    text: "What's on my calendar today?",
    isFinal: true,
    transcribeLanguage: "en-US",
    startTime: 1234567890,
    endTime: 1234567892
  }
}
```

### Step 5: App Processing
The assistant app receives and processes the transcription.

```typescript
// In Assistant App
session.events.onTranscription(async (data) => {
  if (data.text.includes('calendar')) {
    const events = await getCalendarEvents();
    
    // Show each event as a reference card
    for (const event of events) {
      await session.layouts.showReferenceCard({
        title: event.time,
        text: event.title
      });
      await new Promise(resolve => setTimeout(resolve, 3000)); // Show each for 3s
    }
  }
});
```

### Step 6: Display Update
The app sends a display request.

```typescript
// Assistant App → Cloud
{
  type: "DISPLAY_REQUEST",
  packageName: "com.assistant.app",
  view: "MAIN",
  layout: {
    type: "REFERENCE_CARD",
    title: "9:00 AM",
    text: "Team Standup"
  }
}
```

### Step 7: Display Validation
DisplayManager validates and throttles the request.

```typescript
// In DisplayManager
handleDisplayRequest(request) {
  // Check rate limit (200ms between updates)
  if (this.canUpdate()) {
    // Format for glasses
    const formatted = this.formatLayout(request.layout);
    
    // Send to glasses
    this.sendToGlasses(formatted);
  } else {
    // Queue for later
    this.displayQueue.push(request);
  }
}
```

### Step 8: Final Display
The glasses show Alex's calendar.

```typescript
// Cloud → Glasses
{
  type: "DISPLAY_EVENT",
  layout: {
    type: "REFERENCE_CARD",
    title: "9:00 AM",
    text: "Team Standup"
  }
}
```

**Result**: Alex sees their calendar events on the glasses display!

## Scenario 2: Photo Translation

**User Story**: Alex sees a menu in Spanish, takes a photo, and gets it translated.

### Step 1: User Initiates
Alex long-presses the main button.

```typescript
// Glasses → Cloud
{
  type: "BUTTON_PRESS",
  buttonId: "main",
  pressType: "long"
}
```

### Step 2: App Responds
The translator app is subscribed to button presses.

```typescript
// Cloud → Translator App
{
  type: "DATA_STREAM",
  streamType: "button_press",
  data: {
    type: "button_press",
    buttonId: "main",
    pressType: "long"
  }
}

// In Translator App
session.events.onButtonPress(async (data) => {
  if (data.pressType === 'long') {
    // Request photo
    await session.camera.requestPhoto({
      metadata: { reason: 'menu-translation' }
    });
  }
});
```

### Step 3: Photo Capture
PhotoManager handles the photo request.

```typescript
// Translator App → Cloud
{
  type: "PHOTO_REQUEST",
  metadata: {
    reason: "menu-translation"
  }
}

// Cloud → Glasses
{
  type: "PHOTO_REQUEST",
  requestId: "photo-789",
  settings: {
    resolution: "HIGH"
  }
}
```

### Step 4: Photo Return
Glasses capture and send the photo.

```typescript
// Glasses → Cloud
{
  type: "PHOTO_TAKEN",
  photoData: ArrayBuffer,
  mimeType: "image/jpeg",
  timestamp: Date
}
```

### Step 5: Photo Distribution
Cloud routes photo to the translator app.

```typescript
// Cloud → Translator App
{
  type: "DATA_STREAM",
  streamType: "photo_taken",
  data: {
    type: "photo_taken",
    photoData: ArrayBuffer,
    mimeType: "image/jpeg",
    timestamp: Date
  }
}
```

### Step 6: Translation Processing
App processes the image.

```typescript
// In Translator App
session.events.onPhotoTaken(async (data) => {
  // Extract text from image (photoData is ArrayBuffer)
  const text = await ocrService.extract(data.photoData);
  // text = "Pollo Asado - Grilled Chicken\nEnsalada - Salad"
  
  // Translate
  const translated = await translateService.translate(text, 'es', 'en');
  
  // Display results using double text wall
  await session.layouts.showDoubleTextWall(
    `Spanish:\n${text}`,
    `English:\n${translated}`
  );
});
```

**Result**: Alex sees the translated menu on their glasses!

## Scenario 3: Real-time Navigation

**User Story**: Alex is walking and getting turn-by-turn directions.

### Step 1: App Initialization
Navigation app starts and sets up.

```typescript
// On app start
// Subscribe to location updates
await session.location.subscribe();

// Location updates will come through the callback
session.location.subscribeToStream({}, async (data) => {
  // Process location data
});

// Also listen for head position
session.events.onHeadPosition(async (data) => {
  // data.position is "up" or "down"
});
```

### Step 2: Continuous Updates
App receives location updates.

```typescript
// Cloud → Navigation App (periodic updates)
{
  type: "DATA_STREAM",
  streamType: "location_update",
  data: {
    type: "location_update",
    lat: 37.7749,
    lng: -122.4194,
    accuracy: 5,  // meters
    correlationId: "loc-123"
  }
}
```

### Step 3: Smart Display Updates
App updates display based on proximity to turn.

```typescript
let lastUpdateTime = 0;
let nextTurn = null;

// Set up location handler
session.location.subscribeToStream({}, async (data) => {
  const route = await getRoute(data.lat, data.lng);
  nextTurn = route.nextTurn;
  
  // Update more frequently when close to turn
  const distanceToTurn = nextTurn.distance;
  const now = Date.now();
  
  if (distanceToTurn < 50) {
    // Close to turn - update every 2 seconds
    if (now - lastUpdateTime > 2000) {
      await updateDisplay();
      lastUpdateTime = now;
    }
  } else if (distanceToTurn < 200) {
    // Approaching turn - update every 5 seconds
    if (now - lastUpdateTime > 5000) {
      await updateDisplay();
      lastUpdateTime = now;
    }
  }
});

async function updateDisplay() {
  const urgentTurn = nextTurn.distance < 50;
  
  await session.layouts.showReferenceCard({
    title: urgentTurn ? "⚠️ TURN NOW" : "Next Turn",
    text: `${nextTurn.direction} on ${nextTurn.street}\n${nextTurn.distance}m`
  });
}
```

### Step 4: Dashboard Mode
App keeps key info always visible.

```typescript
// Enable dashboard for persistent navigation
await session.dashboard.content.update({
  text: `→ ${nextTurn.distance}m`
});

// Enable always-on dashboard mode
await session.dashboard.mode.set({
  enabled: true,
  alwaysOn: true
});

// Dashboard stays visible even when other apps show content
```

**Result**: Alex gets smart navigation that updates based on their proximity to turns!

## Scenario 4: Multi-App Collaboration

**User Story**: Alex is in a meeting, taking notes while getting real-time transcription.

### Step 1: Multiple Apps Running
Both transcription and notes apps are active.

```typescript
// In UserSession
runningApps: Set(['com.transcription.app', 'com.notes.app'])
```

### Step 2: Shared Event Stream
Both apps receive the same transcription events.

```typescript
// Audio → Transcription → Both Apps
// Cloud → Transcription App
{
  type: "DATA_STREAM",
  streamType: "transcription",
  data: {
    type: "transcription",
    text: "We need to increase our Q4 revenue by 20%",
    isFinal: true,
    transcribeLanguage: "en-US",
    startTime: 1234567890,
    endTime: 1234567895
  }
}

// Cloud → Notes App (same data)
{
  type: "DATA_STREAM",
  streamType: "transcription",
  data: {
    type: "transcription",
    text: "We need to increase our Q4 revenue by 20%",
    isFinal: true,
    transcribeLanguage: "en-US",
    startTime: 1234567890,
    endTime: 1234567895
  }
}
```

### Step 3: Coordinated Display
Apps coordinate through display priority.

```typescript
// Transcription App - Shows real-time text
await session.layouts.showTextWall(
  "We need to increase our Q4 revenue by 20%"
);

// Notes App - Updates dashboard with summary
await session.dashboard.content.update({
  text: "Meeting Notes: 3 items"
});
```

### Step 4: User Interaction
Alex presses button to save important point.

```typescript
// In Notes App
let lastTranscript = '';

session.events.onTranscription((data) => {
  lastTranscript = data.text;
});

session.events.onButtonPress(async (data) => {
  if (data.buttonId === 'main' && lastTranscript) {
    // Save as key point
    await saveKeyPoint(lastTranscript);
    
    // Brief confirmation
    await session.layouts.showTextWall("✓ Saved");
    
    // Clear after 1 second
    setTimeout(() => {
      session.layouts.clearView();
    }, 1000);
  }
});
```

**Result**: Alex seamlessly uses multiple apps together!

## Debugging Tips

### 1. Follow the Message Flow
Use request IDs to trace messages through the system:

```typescript
// Add requestId to track flow
const requestId = `photo-${Date.now()}`;
console.log(`[App] Requesting photo: ${requestId}`);

await session.modules.camera.capturePhoto({
  requestId,
  reason: 'debug-test'
});
```

### 2. Check Event Handlers
Make sure apps have registered handlers for the events they need:

```typescript
// Example: Make sure you're listening for transcriptions
session.events.onTranscription((data) => {
  console.log('[App] Received transcription:', data.text);
});

// For debugging, log all major events
session.events.onButtonPress((data) => console.log('[App] Button:', data));
session.events.onHeadPosition((data) => console.log('[App] Head:', data));
session.events.onPhotoTaken((data) => console.log('[App] Photo received'));
```

### 3. Monitor Rate Limits
Watch for throttled display updates:

```typescript
// In DisplayManager logs
"[DisplayManager] Update throttled - too frequent"
"[DisplayManager] Queued update, will process in 150ms"
```

### 4. Connection State
Always check connection before sending:

```typescript
if (session.isConnected()) {
  await session.layoutManager.showText("Hello");
} else {
  console.error('[App] Not connected!');
}
```

## Common Gotchas

1. **Display Throttling**: Can't update display more than once per 200-300ms
2. **Subscription Timing**: Must subscribe before events occur
3. **Grace Period**: Apps have 20 seconds to reconnect before being stopped
4. **Message Size**: Photos and audio can be large - watch payload sizes
5. **Permission Checks**: Not all users grant all permissions

## Congratulations! 🎉

You now understand how MentraOS Cloud works! You've learned:

- ✅ How authentication and connections work
- ✅ What UserSessions are and how they manage everything
- ✅ The message types that flow through the system
- ✅ How apps connect and use the SDK
- ✅ Real-world scenarios showing complete flows

## Next Steps

1. **Explore the Code**: Dive into the actual implementation
2. **Build a Test App**: Try creating a simple app using the SDK
3. **Read the API Docs**: Check out detailed API documentation
4. **Join the Team Chat**: Ask questions and share ideas

Welcome to MentraOS Cloud development! We're excited to see what you'll build.